Abstract: Taking Into Account Sampling Variability of Model Selection Indices: A Parametric Bootstrap Approach.

AIC and BIC rankings are commonly used in model selection. When the population model is not one of the competing models, this approach leads to selection of a misspecified model and the model selected varies by sample size (Preacher & Merkle, 2012). We propose a parametric bootstrap approach for model selection. A simulation study shows that the proposed approach rejects all candidate models when the population model is not under consideration and picks the population model when it is under consideration. Parametric Bootstrap in Nonnested Model Comparison 3 Taking into account sampling variability of model selection indices: A parametric bootstrap approach In structural equation modeling, researchers often need to compare two or more competing models. For nonnested models, the decision is often made based on the ranking of the Akaike information criterion (AIC) or Bayesian information criterion (BIC). When the set of competing models does not include the population model, Preacher and Merkle (2012) demonstrated that the model selected based on the ranking of BIC varied over repeated sampling and the problem did not diminish with increased sample size. We propose an approach to take into account this sampling variability through parametric bootstrap, which is a extension of Millsap’s (2010) approach to model selection. Parametric Bootstrap The proposed bootstrap approach involves the following steps, using BIC as an example (the same procedure applies to AIC). 1. Fit the competing models (say A and B) to the observed data and save the observed difference in BIC (ΔBIC). 2. Create a set of multiply simulated data (e.g., 1,000) based on the parameter estimates from each of the models. 3. Fit Models A and B to each simulated dataset and save the ΔBIC. This step results in sampling distributions of ΔBIC for model A (ΔBICA ) and ΔBIC for model B (ΔBICB). 4. The observed ΔBIC is then compared to both distributions. If the observed ΔBIC lies in the range (given a priori alpha level) of one of the distributions, then either Model A or B is favored. On the other hand, if the observed ΔBIC lies in the range of both Parametric Bootstrap in Nonnested Model Comparison 4 distributions (suggesting they are equally good) or outside the range of both distributions (suggesting they are equally bad), a selection cannot be determined (Jones & Tukey, 2000). This approach can easily accommodate more than two competing models. See Figure 1 for the example of bootstrap distributions and Table 1 for a summary of decision rules. Table 1. Decision rules of the parametric bootstrap approach. Hypotheses Model A is favored Model A is not favored Model B is favored Two models are equally favored Model B is favored Model B is not favored Model A is favored Neither model is favored Figure 1. Sampling Distribution of ΔBIC under competing models Parametric Bootstrap in Nonnested Model Comparison 5 Table 2. The proportion of each order of the preference models from 1,000 replications in each sample size using the BIC method a . N A > B > C A > C > B B > A > C B > C > A C > A > B C > B > A 80 .500 .485 .005 .000 .010 .000 120 .430 .558 .003 .000 .008 .001 200 .323 .659 .002 .000 .016 .000 500 .173 .793 .000 .000 .034 .00